Skip to content

feat(topology): build the service map from whole-window ClickHouse trace aggregates - #250

Merged
Atom-oh merged 4 commits into
devfrom
feat/trace-graph-aggregate
Nov 10, 2026
Merged

Atom-oh merged 4 commits into
devfrom
feat/trace-graph-aggregate

Conversation

@Atom-oh

@Atom-oh Atom-oh commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Why

The service map's ClickHouse source read the newest 1,000 spans (ORDER BY Timestamp DESC LIMIT 1000). The dev datasource has about 164k spans per hour, so that sample covers roughly the last 20 seconds. A calls edge needs both the parent and the child span inside the sample, so most pairs were lost: low-volume calls and databases dropped out of the map and most spans showed as orphans.

The Prometheus service-graph metrics (traces_service_graph_request_total) were grouped only by client/server. That turned database and queue servers (for example admin, mall, redis) and Tempo's user virtual node into fake service nodes.

What changes

  • ClickHouseOtelAggregateSource (web/lib/trace-source.ts)
    • It reads the table from the catalog's exporter-schema otel_v1 template, or from the unregistered default.
    • It runs three bounded read-only queries over the exact window:
      • attribute groups (service, kind, status, allowlisted resource/span attributes) with span count, summed duration, first/last start, a sample trace/span ID and up to 20 pods
      • parent→child service pairs
      • span-link pairs
    • Each query is limited to 1,000 rows. A saturated limit adds cap_reached. Any failed query fails the whole read, so nodes never publish without their edges.
    • Generated or custom templates and Tempo keep the existing span reader, because their columns are not known to follow the exporter schema.
  • buildTraceGraph (web/lib/trace-graph.ts)
    • It weights aggregate rows. That covers node counts, queries/runs_on/messaging edges, and the instance↔DB observation count and first-seen time.
    • It draws source-counted pairs (evidence: 'trace') as span evidence on the same service nodes, with no self-loops.
  • connection_type is kept in the catalog PromQL (scripts/v2/workers/graph_catalog.py) and parsed in the web code.
    • database and messaging_system rows become db/queue nodes only when no span evidence exists. Spans carry the host-qualified identity and would otherwise duplicate the same database under another name.
    • virtual_node rows draw the uninstrumented side as an external node.
  • graph-store.ts passes the aggregated edges through. graph-sources.ts picks the aggregate source for catalog templates and for the default fallback.
  • Docs (docs/api-reference.md §Trace graph sources) and a bilingual CHANGELOG entry.

Verification

  • npx tsc --noEmit -p . is clean. Full web npm test: 372 files, 5,718 tests passed. pytest scripts/v2/workers/test_graph_catalog.py: 24 passed.

  • New tests cover:

    • template detection, including a rejected ; DROP suffix and non-catalog SQL
    • exact window/limit SQL, and skipping the link query when the table has no Links columns
    • fail-closed handling of partial query failure, cap_reached, malformed rows and empty-proof
    • weights, source-counted edges, and the connection_type node kinds
    • graph-store publication of aggregated edges
  • Live check against the dev ClickHouse (datasource 3, otel.otel_traces, read-only, through awsops-dev-agent-clickhouse-mcp). I ran the exact SQL the code generates:

    Query Rows collectionStatus Time
    attribute groups 196 ok 2.8 s
    parent→child pairs 62 ok 1.3 s
    span links 0 empty 0.9 s

    Fed through the new mapper and builder, those rows produce: status ok, 0 orphan spans, 0 invalid spans, 71 nodes (31 service / 28 workload / 12 db) and 144 edges (62 calls / 54 queries / 28 runs_on). An example edge is wishlist → product-catalog with 2,204 calls.

Rollout notes

  • Web: Deploy Web on dev picks up the aggregate source. With no registry rows on dev (datasource_diagnosis_enabled=false), the default fallback is the aggregate source. It relies on the datasource's configured database, which is otel on dev.
  • Worker: the graph_catalog.py change ships in the worker Lambda zip, so it needs the usual Terraform plan/apply. Catalog rows pick up connection_type on the next datasource_index run. Until then the old PromQL simply lacks the label and behaves as before.

🤖 Generated with Claude Code

…ace aggregates

The ClickHouse trace source joined the newest 1,000 spans of a ~160k-span hour, so
most parent/child pairs fell outside the sample: low-volume calls and databases
dropped out and most spans showed as orphans.

- ClickHouseOtelAggregateSource reads the catalog's exporter-schema table with three
  bounded read-only queries: service/kind/status attribute groups with span counts,
  parent->child service pairs and span-link pairs (1,000-row limit each; a saturated
  limit is cap_reached, any failed query fails the read). Generated/custom templates
  and Tempo keep span sampling.
- buildTraceGraph weights aggregate rows and draws source-counted pairs as span
  evidence on the same service nodes.
- Service-graph metric rows keep connection_type: database/messaging servers become
  db/queue nodes only without span evidence, and virtual_node rows draw the
  uninstrumented side as an external node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

🤖 AI Code Review (two independent reviewers)

Reviewer responses: codex/ALL claude/ALL

Status: PASSED — No blocking issues found

All verification is done. Both panel claims are now reconciled against BASE + patch; I can write the chair synthesis.

Chair Synthesis — PR #250: whole-window ClickHouse trace aggregates

1. Summary

The PR replaces the newest-1,000-span sample with three bounded read-only ClickHouse aggregate queries (node groups, parent→child pairs, span links) for catalog exporter-schema sources, adds weighted span rows and source-counted call edges to the graph builder, rewrites node-cap selection to keep edge endpoints together, and filters Tempo service-graph rows by connection_type. The panels disagreed sharply — codex reported zero findings in a four-sentence report, claude reported six MAJORs — so the chair independently re-verified every gate-relevant claim against BASE and the patch. The implementation is sound and strictly read-only; all verified findings are quality/disclosure issues that are either documented design tradeoffs or affect only internal cap-ranking, none rising to gate-blocking severity.

Chair verification performed: graph_catalog.py:_clickhouse_trace_spans emits exactly the template CATALOG_OTEL_TEMPLATE + EXPORTER_COLUMNS accept (optional columns appended after SpanAttributes, per-part backtick-quoted table), so the aggregate path genuinely activates for catalog rows — the "feature silently inert" risk is excluded. The generated SQL passes sql_readonly_guard.assert_read_only end-to-end: first token SELECT, no semicolons, 'db.system'/'messaging.system' literals are stripped before the SYSTEM/SET/INTO DANGER words match, no function trips _TABLE_FN (\b-anchored prefixes; mapFilter/startsWith/fromUnixTimestamp64Milli etc. are safe), no SETTINGS clause. timestamp() accepts numeric epoch-ms, so aggregate rows map correctly. ${shape.table} can only come from the anchored regex's identifier group and cannot break quoting.

2. Issues per lens

L2 — Code correctness

No CRITICAL/MAJOR adopted.

  • MINOR — metric self-call rows double node metricCount and draw a self-loop (web/lib/trace-graph.ts, metric loop). One model (claude) flagged MAJOR; codex found nothing. Chair verified the mechanics (when client === server resolves to one node id, metricWeight runs twice and a self-loop edge is drawn) but downgrades: the self-loop metric edge is pre-existing BASE behavior, node metricCount is rendered nowhere in web/components (verified by grep — it feeds only volume() cap-ranking in graph-store.ts), and 2× is internally consistent with the "count per edge endpoint" accounting used for ordinary A→B edges. The asymmetry with the trace-evidence branch (if (client !== server)) is still worth aligning.
  • MINOR — node-cap boundary can admit a dependency without its caller (one model flagged). Verified: loop 1 skips (doesn't break on) a 2-slot edge with 1 slot left; loop 2 then fills by volume regardless of kind, so an isolated db/queue node can publish with its edge filtered away. Benign (drops force partial) but contradicts the doc wording (see L5).
  • MINOR — any failed aggregate (including the supplementary links query) discards all three and falls back to sampling (one model flagged). Deliberately conservative per the in-code rationale ("never publish nodes without their edges") and disclosed as incomplete_collection; treating a links-only failure as partial on an otherwise-complete aggregate read would be strictly better.
  • MINOR — DEFAULT_OTEL_SHAPE assumes links: true for never-introspected sources (one model flagged). On an exporter table without Links.*, every rebuild wastes one failing query and degrades to sampling. Degradation is disclosed, but defaulting links: false is safer.

L3 — Security / AWS mutation safety

No findings — both models agree, and the chair independently confirmed. Every new statement is a SELECT; no IAM, Terraform resource, or mutation surface is touched (CATALOG_VERSION = "v4" is a cache-invalidation constant). ADR-005 is not implicated. Injection surfaces are closed: endMs gated by Number.isSafeInteger before interpolation, windowMins/limit floored/clamped, shape.table regex-constrained, and the Lambda-side guard verified above as defense-in-depth. The ClickHouse connector's lexical guard being the primary barrier there (per agent/lambda/CLAUDE.md) makes claude's two MINOR suggestions worth adopting: a producer-contract test feeding aggregateSql's three strings through the Python guard (a future edit that trips the guard would silently and permanently degrade the feature to sampling), and an explicit tighter max_execution_time for the two join queries (invocations per rebuild triple, and the self-join's scan cost is bounded only by the connector's time budget).

L4 — Observability / data-integration correctness

No CRITICAL/MAJOR adopted. One model (claude) flagged five MAJORs here; codex found nothing. The chair verified each claim's facts but downgrades all five on severity:

  • MINOR — group-key cardinality scales with EC2 instance count. Factually correct (cloud.resource_id/host.id/service.instance.id EC2 values are kept verbatim), but keeping them is functionally required — observedInstanceDatabasePair (verified in BASE trace-graph.ts:78-127) consumes exactly those claims to build instance_observations. Overflow is disclosed (ORDER BY Weight DESC LIMIT 1000 + cap_reached → partial), and coverage remains strictly better than BASE's newest-1,000-span recency sample. A split instance-claim aggregate is a good follow-up, not a defect.
  • MINOR — service.version in AGG_RESOURCE_KEYS is pure cardinality waste. Chair-verified: serviceVersion is written by spanMetadata and consumed by nothing — not scope() (BASE trace-graph.ts:39-45), not node meta, no component. Remove it from the group key; it only worsens cap pressure.
  • MINOR — nonServiceMetricCalls is computed and never published. Chair-verified: the patch adds it to buildTraceGraph's return but not to details, the graph-state.ts:62 projection allowlist, or COLLECTION_LOSS_KEYS. However, the drop it counts is intentional and documented ("are not drawn"), so this is missing disclosure plumbing / a dead field, not incorrect published data. Plumb it through or delete it.
  • MINOR — window-boundary parent spans lose the former orphanSpans signal. Real, but explicitly documented in docs/api-reference.md ("not counted as a pair and is not disclosed"), inherent to any windowed aggregate, small relative to a 60-minute window — and the BASE orphan counter was dominated by exactly the sample-truncation noise this PR exists to remove.
  • MINOR — mixed deployments rank whole-window counts against ≤1,000-span samples in the node/edge caps, so a busy aggregate source can crowd out Tempo nodes. Documented ("a graph mixing them sums both kinds of count"), and over-cap graphs publish as partial; a per-source budget would be a worthwhile follow-up.
  • MINOR — pod-list truncation (WORKLOAD_POD_CAP=50, groupUniqArray(20)) is undisclosed, unlike instance_observations_omitted. BASE had no cap at all, so this is a net tightening; add a truncation marker for consistency.
  • The connection_type catalog change is verified backward-compatible (absent label → no extra series; text('') → undefined keeps plain rows), istio_v1 correctly unchanged, and the CATALOG_VERSION v4 bump makes the CHANGELOG rebuild note accurate.

L5 — Docs / ADR consistency

No CRITICAL/MAJOR. No ADR bodies exist in this tree to diff (only ADR-023); the mutation-freeze invariant is untouched. CHANGELOG follows the repo rule: one feature bullet per section, no PR/round numbers, EN/KO 1:1. New runbook/api-reference content is English-only as required. Verified-accurate: caps (200/500/1,000), query counts, reason taxonomy, exclusion wording, and the runbook's test command and cross-link anchor.

  • MINOR — docs/api-reference.md overstates "a kept database, queue or workload always keeps its caller" (one model flagged; chair confirmed the boundary counterexample in L2). Soften the wording or fix the fill loop to match it (preferred).
  • MINOR — stale JSDoc on InstanceObservation: "correlation_id names that span" is false for aggregate rows (max(Timestamp) vs any(SpanId)); the api-reference got the correcting sentence, the code comment didn't.
  • MINOR — sampledDurationMs now holds a whole-window sum for aggregate sources; neither doc section mentions it and the name misleads.
  • MINOR — README's graph_querygen_enabled description ("the ONE ClickHouse trace_spans graph query") is now imprecise since the catalog template and generated templates take different readers.

3. Suggestions

  1. Align the metric self-call path with the trace branch: compute endpoints first and either skip self-loops or apply metricWeight once per distinct node.
  2. Drop service.version from AGG_RESOURCE_KEYS; consider a separate capped aggregate for instance claims so node/db coverage no longer shares its cap with per-instance cardinality.
  3. Either plumb nonServiceMetricCalls through details → projection → COLLECTION_LOSS_KEYS (+ four locale labels) or remove the field.
  4. Add a producer-contract test running aggregateSql's three statements through sql_readonly_guard/_TABLE_FN, and pass a tighter max_execution_time on the two join queries; document the per-rebuild invoke budget as TempoTraceSource does.
  5. Treat a failed links-only query as incomplete_collection instead of discarding two good aggregates; default un-introspected shapes to links: false.
  6. Fix the three stale/overstated doc spots (JSDoc, "always keeps its caller", sampledDurationMs) and the README graph_querygen_enabled line.

4. Verdict

Codex's clean report and claude's findings converge on the only gate-relevant questions — read-only safety and the absence of missing-definition/activation defects — which the chair independently confirmed. Claude's six MAJORs are each factually grounded but, on chair verification against BASE and the resulting code, all land as disclosed design tradeoffs, internal-ranking effects, or missing-disclosure plumbing rather than incorrect published data, security exposure, or policy violations. No CRITICAL or MAJOR finding survives verification; the image manifest lists no images.

IMAGE_COVERAGE: NOT_REQUIRED


Triggered by commit 891df138ff7c2aef8c03ff612ed0d7a1d2fa1760 · workflow: .github/workflows/pr-review.yml

- Bump graph CATALOG_VERSION to v4 so indexed Prometheus/Mimir instances rebuild
  and pick up the connection_type PromQL.
- Defer a metric database/queue row only when trace spans show the same client
  service using a database/queue, instead of whenever any span exists; count
  deferrals as deferredMetricCalls.
- Accept the aggregate path only for exactly the bare exporter columns, so
  generated or aliased otel_v1 templates keep the span reader.
- Push the sorted identity map into both join sides (bounded hash build) and
  mapSort grouped maps so equal attribute sets group together.
- Keep ARN service.instance.id claims so claimedInstance can still veto them.
- Require empty proof from every aggregate; guard link-array lengths before
  ARRAY JOIN; sum durations in milliseconds within safe integers.
- Metric-derived db nodes carry no account/region attribution; metric volume
  ranks nodes under the node cap with external peers ranked like services.
- Document the two-plus-optional-links queries, caps and boundary pairs, and the
  runbook aggregate contract; add an external node color.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…coped

- Reduce host.id/service.instance.id to EC2 IDs/ARNs and any other
  cloud.resource_id to one 'unbindable' marker, so host UUIDs stop splitting
  groups while claimedInstance keeps its cloud.resource_id veto.
- Treat a metric database/queue row as a duplicate only when a span from a
  compatibly scoped client of the same name names that dependency; other
  dependencies and scopes keep their metric rows. The internal counter is no
  longer documented as published metadata.
- Rank label-only metric db/queue nodes by volume with services so a
  high-cardinality metric cannot evict service nodes.
- Clamp aggregate rows to the limit, disclose out-of-window rows, cap merged
  workload pods and bound metric node names.
- Docs: distinguish failed envelopes from skipped rows, qualify counts by
  cap_reached, and move the mocked suites out of the PostgreSQL block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…des by edge, fall back to sampling

- Service-graph metrics contribute service->service calls only, as graph-store's
  metrics-source contract already states. Rows Tempo marks database,
  messaging_system or virtual_node are not drawn (they are not services, and their
  label-only nodes would carry no provenance through the agent SQL reader view);
  databases and queues come from trace spans with host-qualified identity. This
  removes the metric db/queue/external node kinds and the span-based deferral.
- Trace node-cap selection takes the heaviest edges' endpoints first, so a kept
  db/queue/workload always keeps its caller and many low-volume dependencies cannot
  evict their services.
- A failed aggregate query falls back to the bounded newest-span reader for that
  rebuild (partial, incomplete_collection) instead of retaining a frozen graph; a
  malformed envelope still fails the read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Atom-oh
Atom-oh merged commit cc7586f into dev Nov 10, 2026
6 checks passed

This branch was previously deployed

1 inactive deployment
ci-review-auto — 891df138 Deployed Nov 10, 2026 by Atom-oh via AI Code Review #996
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant